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Abstract: Large datasets with interactions between objects are common 
to numerous scientific fields (i.e. social science, internet, biology...). The 
interactions naturally define a graph and a common way to explore or 
summarize such dataset is graph clustering. Most techniques for clustering 
graph vertices just use the topology of connections ignoring informations 
in the vertices features. In this paper, we provide a clustering algorithm 
exploiting both types of data based on a statistical model with latent struc- 
ture characterizing each vertex both by a vector of features as well as by its 
connectivity. We perform simulations to compare our algorithm with exist- 
ing approaches, and also evaluate our method with real datasets based on 
hyper-textual documents. We find that our algorithm successfully exploits 
whatever information is found both in the connectivity pattern and in the 
features. 



1. Introduction 

Classical data analysis has been developed for sets of objects with features, but 
when explicit relationships exist between objects, classical data analysis can- 
not take these relations into account. On the other hand, much recent research 
has been performed for analyzing graphs, for example in finding relationships 
in social sciences, gene interactions in biology and hyperlinks analysis in com- 
puter science, providing insights into the interactions in these networks. Many 
approaches to graph analysis have been proposed. Model-based approaches, i.e., 
methods which rely on a statistical model of network edges and vertices, such as 
those first proposed by Erdos-Renyi , often allow to get insight into the network 
structure deducing their internal properties. 

An interesting alternative to the basic Erdos-Renyi model which docs not 
fit well to real networks is to consider a mixture of distributions (Frank & 
Harary, 1982; Snijders & Nowicki, 1997; Newman & Leicht, 2007; Daudin ct al., 
2008) where it is assumed that nodes are spread among an unknown number 
of latent connectivity classes. Conditional on the hidden class label, edges are 
still independent and Bernoulli distributed, but their marginal distribution is a 
mixture of Bernoulli distributions with strong dependence between the edges. 
Many names have been proposed for this model, and in the following, it will 
be denoted by MixNet, which is equivalent to Block Clustering of Snijders & 
Nowicki (1997). Block-Clustering for classical binary data can be dated back to 
early work in the seventies (Lorrain & White, 1971; Govaert, 1977). 

But vertex content is also sometimes available in addition to the network 
information used in the methods mentioned above. A typical example is the 
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world-wide-web which can be described by either hyperlinks between web pages 
or by the words occurring in the web pages: each vertex represents a web page 
containing the occurrences of some words and each directed edge a hyperlink. 
The additional information represented by the vertex features is rarely used in 
network clustering but can provide crucial information. Here we combine infor- 
mation from both vertex content traditionally used in classical data analysis 
to information found in the graph structure, in order to cluster objects into 
coherent groups. This paper proposes a statistical model, called CohsMix (for 
Covariates on hidden structure using Mixture models), which considers the de- 
pendent nature of the data and the relation with vertex features (or covariates) 
in order to capture a hidden structure. 

Considering spatial or relational data neighbourhood is not an original ap- 
proach in clustering. For instance, Hidden Markov Random Fields (HMRF) 
are well adapted to handle spatial data and are widely used in image analysis. 
When the spatial network is not given it is generally obtained using Dclaunay 
triangulation (Ambroise et al., 1997). 

Hoff (2003) proposed a new way to deal with covariates. He suggested to 
model the expected value of the relational ties by a logistic regression. The prob- 
lem of this method is the dependence between the observations conditional on 
the regression parameters and the covariates. Hence, he proposed to incorporate 
random effect structures in a generalized linear model setting. The distribution 
of dependence among the random effects determines the dependence among the 
edges. 

There are also approaches based on non statistical frameworks. In particular, 
it is noticeable that there exits a strong similitude between multiple view and 
graph models with covariates. In fact, multiple view learning algorithms (Ruping 
& Scheffer, 2005) consider instances which have multiple representations and 
simultaneously exploit these views to find a consensus partition. 

The second section introduces the proposed model, which is an extension 
to the MixNet model. Since the model considers a great number of dependen- 
cies, the proposed estimation scheme proposes a variational approach of the 
EM algorithm. This approach allows us to deal with larger network than the 
Bayesian framework. Then we introduce practical strategies for the initialization 
and the choice of the number of groups. In the third section extensive simula- 
tions illustrate the efficiency of the this algorithm and real datasets dealing with 
hyper-textual documents are studied. A R package named CohsMix is available 
upon request. 

2. A Mixture of Network with covariates 

This section introduces the proposed model. We choose to consider a model 
which assumes independence of covariates and edges conditional on the node 
classes. It assumes that both the connectivity pattern and the vertex features can 
be explained by the class. In the web context this model considers that a given 
class contains documents which have both a similarity between occurring words 
and a similarity of connectivity pattern with documents inside and outside the 
class. Although this assumption does not explicitly model the idea that authors 
tend to link similar topics (occurring words) which creates a thematic locality 
(Davison, 2000), it allows us to detect clusters of local theme. Its simplicity 
makes it a robust well adapted model to the real web. 
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2.1. Models and Notation 



Let us define a random graph G, where V denotes the set of vertices. Based 
on the MixNet model, our model assumes that V is partitioned into Q hidden 
classes. Let us denote by Zt q the indicator variable such that {Zi q — 1} if node 
i belongs to class q. Z = (Zi,...,Z„) is the vector of random independent 
indicator variables such that 



Zi~M{l,a = {a U -,aiQ}), (1) 
with a the vector of class proportions. Edges are Bernoulli random variables 

Xij\Z iq Zji = 1 ~ B(ir ql ), (2) 
conditionally independent, given the node classes 

P(X|Z) = l[l[p(x ij \z iq z jl = l) z « z ». 

ij q-i 

In this paper, we consider an undirected graph we suppose that there is no 
self-loops, i.e. a node can not be connected to itself (Xu = 0). Nevertheless, the 
method can easily be generalized to encompass directed graphs with self-loops. 



Vertex Features. Hereafter we consider n objects described both by their 
connections and p features. In that case the data under study can be repre- 
sented into different forms. One might for example consider a two part vector 
for characterizing each object, where the first part contains the feature of the ob- 
ject Yj and the second part contains a binary vector representing the connection 
to all n — 1 other objects Xj. Continuing our example about world- wide- web, 
the web pages can be viewed as a vector of word occurrences with hyperlinks 
or as two matrices. One based on the adjacency matrix describing the topology 
of the graph generated by the hyperlinks and the other by the features matrix 
generated by the word occurrences in each web page. 

Hereafter we consider that the p dimensional feature vector associated to 
object i is defined by : 



(2) 



Y; 



We assume that the feature vectors Y, are multivariate normally distributed 



Yi\Z iq = 1 ~ .VOv E q ) 



(3) 



where 
identity. 



( \ 



and S q = a I the covariance matrix is proportional to the 
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Fig 1. Graphical representation of the CohsMix Model. The squares represent discrete random 
variables and circles continuous random variables. 



The random feature vectors Yj are conditionally independent, given the node 
classes 

P(Y\Z) = Hl[P(Y l \Z tq ) z ^. 

i q 

The conditional distribution associated to covariates can be written as follow : 
logP(Y|Z) = ]T]TZ 19 log P(Y, | 



= EE** 



The proposed mixture model assumes independence of X and Y conditional 
on Z. Considering this independence between edges and covariates, the complete 
log- likelihood can be written as (Figure 1): 

P(X,Y, Z) = P(Z)P(X, Y|Z) = P(Z)P(Y|Z)P(X|Z). 

The next section proposes an estimation scheme for the CohsMix model. 



2.2. Variational EM algorithm for CohsMix 

In the classical em framework developed by Dempster ct al. (1977), where X 
and Y are the available data, the inference of the unknown parameters © spread 
over a latent structure Z uses the following conditional expectation: 



q(&\& 



(rn)\ _ 



E 



{log£ c (X,Y,Z;0)|X,Y;0(™)} 

= ^ P (Z|X,Y; ©M) log £ c (X,Y,Z;0) (4) 



zez 
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where 

@(m+i) = ArgmaxQ(0, 0^). 
© 

The usual em strategy would be to alternate an E-step computing the condi- 
tional expectation (4) with an M-step maximizing this quantity over the param- 
eter of interest 0. Unfortunately, no closed form of Q (0|0( m )) can be formu- 
lated in the present case. The technical difficulty lies in the complex dependency 
structure of the model. Indeed, P(Z|X,Y;0) cannot be factorized, as argued 
in Daudin ct al. (2008). This makes the direct calculation of Q (0|©( m )) im- 
possible. To tackle this problem we use a variational approach (see, e.g., Jordan 
ct al., 1999, for elementary results on variational methods). In this framework, 
the conditional distribution of the latent variables P(Z|X, Y; 0( m )) is approx- 
imated by a more convenient distribution denoted by i?(Z), which is chosen 
carefully in order to be tractable. Hence, our EM-like algorithm deals with the 
following approximation of the conditional expectation (4) 

E fl {log£(X,Y,Z;0)}= £ R(Z) log £(X, Y, Z; 0). (5) 

zez 

In the following section we develop a variational argument in order to choose 
an approximation i?(Z) of P(Z|X, Y; 0('™'). This enables us to compute the 
conditional expectation (5) and proceed to the maximization step. 

2.3. Variational estimation of the latent structure (E-step) 

In this part, is assumed to be known, and we are looking for an approximate 
distribution R(-) of the latent variables. The variational approach consists in 
maximizing a lower bound J of the log- likelihood logP(X,Y;0), defined as 
follows: 

J(&) = logP(X,Y;0) ~B KL {i?(Z)||P(Z|X,Y; ©M)} (6) 

where T>kl is the Kiillback-Leibler divergence. This measures the difference 
between the probability distribution P(-|0) in the underlying model and its ap- 
proximation i?(-). An intuitively straightforward choice for R(-) is a completely 
factorized distribution (see Mariadassou & Robin, 2007; Zanghi ct al., 2008) 

R(Z) = ]Jh Ti (Z i ), (7) 

iev 

where h Ti is the density of the multinomial probability distribution M.(l;Ti), 
and Ti = (tjx, ■ • ■ , TiQ) is a random vector containing the variational parameters 
to optimize. The complete set of parameters r = {Ti q } iGV q( -Q is what we are 
seeking to obtain via the variational inference. In the case in hand the variational 
approach intuitively operates as follows: each r iq can be seen as an approxima- 
tion of the probability that vertex i belongs to cluster q, conditional on the data, 
that is, n q estimates F(Z iq = 1|X, Y; 0), under the constraint ^2 q T iq = 1. In 
the ideal case where P(Z|X,Y;0) can be factorized as P(Zj|X, Y; 0) and 
the parameters Ti q are chosen as Ti q = P(Zi q — 1|X, Y; 0), the Kiillback-Leibler 
divergence is null and the bound J reaches the log-likelihood. 
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The lower bound J to be maximized in order to estimate r can be expressed 



as 



Jt = Er.(z) {J(&)} = E fl(z) {log(P(X, Y, Z))|X, Y; ©} - £\R(Z) log(i?(Z)). 

z 

The optimal approximate distribution R is then derived by direct maximiza- 
tion of J7t-. Let all the parameters jr q i,a q ,fl q and a be known. The follow- 
ing fixed-point relationship holds for the optimal variational parameters f — 
arg max T J T . 



f (m+ 
iq 



k=l 



(8) 

Once again, the maximization of ^J T provides the optimal values of the parame- 
ters. The optimal parameters a q ,TTqi,fJ, q and <r, i.e. the parameters maximizing 
J T satisfy the following relations: 



Vq 



1 

i—l 

i¥=3 



i 

E 7 ^ 



and a 



(9) 



^^r 49 (Y i -C 9 ) T (Y l -C,) 

i 1 



For completeness, we summarize the variational EM algorithm for CohsMix in 
the Algorithm 1. 



2-4- Model selection: ICL algorithm 



As the number of clusters is an unknown parameter of our statistical model, 
it is possible to use the Integrated Classification Likelihood (ICL) to choose 
the optimal number of classes (Bicrnacki et al., 2000). The ICL criterion is es- 
sentially derived from the ordinary BIC considering the complete log-likelihood 
instead of the log- likelihood. This optimal number is obtained by running our 
algorithm concurrently for models from 2 to Q classes and selecting the solution 
which maximizes the ICL criterion. In our situation where additional covariates 
are considered, the ICL criterion can be written as: 

ICL(Q) = maxlog£(X, Y, Z; Q, Q) - l - x Q(Q - 1) log( " ( "~ 1} ) - log 

v v ' v v 

related to ir q i related to 

-p(p-l)log( 2 ; )+pxQlog( 2 ' ) 

" v ' 

related to fi q and cr 
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Algorithm 1: Variational EM CohsMix Algorithm 

Data: Matrices of connectivities X and similarities Y 
/* Initialization of the parameters */ 

BW^af,..,^,^,..,^,^ ^ ie <«),m = 

while not convergence do 
/* Estimation step */ 

/* Compute t = {riq\ the probabilities that vertex i belong to 
cluster q finding fix point of g() */ 
foreach i £ {1, N} do 

foreach q £ {1, Q} do 
I r^ m+1) = g(x( m ') (see Equation 8) 

/* normalize posterior probabilities */ 

scale = T «? 

L ^ = ^^k' v «e {i,..., Q} 

/* Maximization step */ 

/* re-estimate the distribution parameters to maximize the likelihood of 
the data */ 

Update parameters according Equation 9 : 
foreach q £ {1, Q} do 

4 m+1) =Arg m ax aa> 7 T (0) 
foreach I £ {1, Q} do 

tt(™ +1) =Argmax^ i ^(©) 

^ m+1) =Argmax M(j ^(0) 
m = m +1 

Result: Estimated parameters © and posterior probabilities Ti q 
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Experiments 


Q 


nbCov 


d(\,e) 




a 


{2, ...,12} 


3 


0.4 


4 


b 


5 


{2, ...,15} 


0.2 


4 


c 


3 


3 


{0,...,0.5} 


4 


d 


3 


3 





{4,..., 8.5} 



Table 1 

Parameters of the four different settings which are used to generate the J l 3 affiliation models 

considered in the experiments. 



This expression of the ICL criterion is based on the method described in Daudin 
et al. (2008). 

3. Experiments 

In this section, we report experiments in order to assess the performances and 
limitations of the proposed model in a clustering context. We consider synthetic 
data generated according to the assumed random graph model, as well as real 
data from the web. Using synthetic graphs allows us to evaluate the quality of 
the parameter estimation. In parallel, we also compare classification results with 
alternative clustering methods using a ground truth. The real datasets consist 
of hypertext documents retrieved from a websearch query. A R package named 
CohsMix is available upon request. 

3.1. Comparison of algorithms 

Simulations set-up In these experiments, we consider simple affiliation mod- 
els with two parameters defining the probability of connection between nodes 
of the same class and of different classes, respectively Tt qq — A and -K q i = e and 
equal mixture proportion a.\ = ... = ctQ = ^. We consider models with n = 150 
nodes. 

We generate graph models in order to evaluate the algorithm performances 
as the difficulty of the problem varies. The clustering problem increases in dif- 
ficulty with the number of classes Q, the number of features nbCov, d(X, e) 
the euclidean distance between intra and extra connectivity parameters and 
d(fi q , fn) the distance between the feature mean vectors of classes. We decide to 
focus on these parameters to produce data with different levels of structure and 
eventually consider 43 different graph models whose description are summarized 
in Table 1. Each model is simulated 20 times. 

We use the adjusted Rand Index (Hubert & Arabie, 1985) to evaluate the 
agreement between the estimated and the actual partition. The Rand index is 
based on a ratio between the number of node pairs belonging to the same and 
to different classes when considering both partitions. It lies between and 1, 
two identical partitions having an adjusted Rand Index equal to 1. 

To avoid initialization issues, the algorithm is started with multiple initial- 
ization points and the best result is selected based on its likelihood. Thus, for 
each simulated graph, the algorithm is run 10 times and the number of clusters 
is chosen using the Integrated Classification Likelihood criterion, as proposed 
in the previous section. 
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Alternative clustering methods Additionally to the CohsMix algorithm 
study, we compared it with two "rivals" : a multiple view learning algorithm 

(Ruping & Scheffcr, 2005; Zhang et al., 2006), and a Hidden Markov Random 
Fields (Ambroise et al., 1997): 

• Spectral Multiple View Learning (SMVL) : There exits a strong simili- 
tude between multiple view and graph models with covariates. In fact, 
multiple view learning algorithms consider instances which have multiple 
representations and simultaneously exploit these views to find a consensus 
partition. This is achieved via spectral clustering on a linear combination 
of a standard kernel corresponding to the graph structure and a kernel 
corresponding to vertex proximity. 

• Hidden Markov Random Fields (HMRF) : Hidden Markov Random Fields 
are commonly used to handle spatial data and are widely used in image 
analysis. We use a classical Potts model on the latent structure which 
encourage spatial smoothing of the cluster. This kind of approach uses 
the graph structure to smooth the partition of the vertex over the graph, 
whereas the approach proposed in this paper uses the graph structure 
directly to estimate the vertex partition. 

Simulations results We focus our attention on the Rand Index for each 
algorithm. Indeed, a well estimated partition leads to good estimates. 

As expected, the performance of the three algorithms decrease with the num- 
ber of groups (Figure 2 a) . 

A first interesting result is that, in presence of a modular structure (Figures 2 
a,b and c) in the network and weakly informative features, CohsMix algorithms 
always performs better than SMLV and HMRF algorithms. 

Besides, it is noticeable that performances of CohsMix increase with the num- 
ber of features and/or with the distance between mean vectors (Figure 2 b and 
d). HMRF algorithm with Potts a priori use the neighborhood structure for 
smoothing the partition. A vertex with all its neighbors of a given class has a 
high probability to be assigned to this class but HRMF does not take advantage 
of the graph structure as fully as Cohsmix. Our model is thus mainly attractive 
and suitable for datasets with an existing graph structure. 

When there is no graph structure at all and few informative features (Figure 
2d) the Cohsmix does not compare to HMRF or SMLV. The Cohsmix algorithm 
is more sensitive to the total absence of graph structure than its competitor. 

But in all other setup, the quality of partition estimation remains good with 
different kind of models, the CohsMix algorithm appears very attractive and 
suitable for structured graphs with vertex features. We shall see in the next 
section that this algorithm also performs well on real web datasets. 

3.2. Real data 

Exhaustivity is an essential feature for information retrieval systems like Web 
search engines. However, it appears that ambiguous queries produce such a 
huge diversity in the responses that it is a real impediment to understanding. 
A common way to circumvent this situation is to organize search results into 
groups (clusters), one for each meaning of the query. This concern has been 
in the focus of the information retrieval community (Hearst & Pedersen, 1996; 
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0.1 0.2 0.3 0.4 0.5 4 6 8 10 12 14 16 

Distance between intra and inter connectivity Distance between mean vectors 



(c) (d) 

Fig 2. Comparison of HMRF, Spectral MLV and CohsMix. (a) Varying Q the number of 
classes, (b) Varying the number of Features, (c) Varying the distance between intra and inter 
connectivity parameters, (d) Varying the distance between the mean vector of the classes. 

Zamir & Etzioni, 1998) since the early days of the Web. More recently, academic 
(Zeng et al., 2004) and industrial (Bertin & Bourdoncle, 2002) (exalead.com 
or clusty . com) attempts have made the clustering of search results a common 
feature for a WWW user. 

The main drawback of many Web page clustering methods is that they take 
into account only the topical similarity between documents in the ranked list 
and they do not consider the topology induced by hyperlinks. But in competitive 
or controversial queries like "abortion" or "Scientology" such methods do not 
reveal community information that is visible on the link topology : By affinity, 
authors tend to link to pages with similar topics or points of view which create a 
thematic locality (Davison, 2000). In addition, ambiguous queries like "orange" 
or "jaguar" can also benefit from the link topology to produce more accurate 
separation of results. Combination of topological and topical clustering methods 
is a proven strategy to build an relevant system. One of the most relevant 
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previous work is suggested in He et al. (2002), which build a Web page clustering 
system which accounts for the hyperlinks structure of the Web, considering two 
Web pages to be similar if they are in parent/child or sibling relations in the Web 
graph. A more general multi-agent framework based on path between each pair 
of results has been proposed by Bckkcrman et al. (2006), but these methods, 
not model-based, use various heuristics and fine tunings. 

Datasets setup We use exalead.com search engine in our real data experi- 
ments. For each query, we retrieve the first 150 search results in order to build 
our graph and feature structures. Indeed, the web is a very sparse graph and the- 
matic subgraphs may amplify this property creating unconnected components 
which inhibits the opportunity to use classical graph clustering algorithms di- 
rectly on the observed adjacency matrix. In order to increase the graph density, 
the probability to have a link between two nodes, we propose to use the site 
graph of exalead.com basically based on the concepts of Raghavan & Garcia- 
Molina (2003). In this graph, nodes represent websites (a website contains a 
set of pages) and edges represent hyperlinks between websites. Multiple links 
between to different website are collapsed into a single link. Intra-domain links 
are taken into account if hostnames/ websites are not similar. This site graph is 
previously computed. This methodology is similar to the Exalead application 
called Constellations : constellations.labs.exalead.com. 

Then, text features are extracted from the content of the web page returned 
by the search engine. The features are built using various text processing like 
normalization, tokenization, entities detection, noun phrase detection and re- 
lated terms detection. Besides, we remove rare features which do not appear 
more than twice. Eventually each feature vector is approximately of dimension 
p = 100 and summarizes all text of a returned page. 

Algorithm results. We choose one ambiguous query ("jaguar") and one con- 
troversial query ("Scientology") to illustrate our algorithm behavior with real 
datasets. In Figure 3 associated to the query "Scientology", we can observe 
a well structured graph which fits our estimated latent partition with an op- 
timal number of classes Q = 3. Basically this partition yields the pro- and 
anti-Scientology clusters and identifies a gateway cluster (composed for exam- 
ple by http://en.wikipedia.org/wiki/Scientology) bridging the pro and 
anti cluster. Then, we concentrate our attention on the most representative text 
features of each class q. To succeed, we select the best occurrence of term fea- 
tures in the different [i q . Once again (see 3) , we notice terms describing pro- like 
("self esteem or "providing real solutions") and anti- like ("criticism of dianet- 
ics" or "truth about Scientology"). The interface class is composed by common 
terms describing the church of Scientology. Thus, in a web context, the CohsMix 
algorithm is enable to named the different found partitions which a precious 
assistance to have rapidly a global overview of the hidden structure. 

The results of the processing of the ambiguous query "jaguar" is represented 
in Figure 4. CohsMix clearly identifies three contexts: computer, animal and car 
model related web pages. 

The above results illustrate that our algorithm CohsMix seems well adapted 
to detect ambiguous or controversial queries of WWW search engine users. 
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Fig 3. Representation of the results of a clustering of the webpages returned by the contro- 
versial query "Scientology" using CohsMix. The graph structure is represented on the left and 
on the right are the main features. Colors indicate the CohsMix classification 
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Fig 4. Representation of the results of a clustering of the webpages returned by the ambiguous 
query "jaguar". 



4. Conclusion 



This paper has proposed an algorithm for clustering dataset whose modelisation 
could be a graph structure embedding vertex features. Characterizing each ver- 
tex both by a vector of features as well as by its connectivity, CohsMix algorithm, 
based on a variational approach of EM, uses both elements to cluster the data 
and estimate the model parameters. When analyzing simulation and compari- 
son results, our algorithm appears very attractive and competitive for various 
kind of models. We have tested CohsMix algorithm to cluster web search results 
based on hypertextuality and content and we demonstrate good relevance of 
this model approach. We find that our algorithm successively exploits whatever 
information is found both in the connectivity pattern and in the features. In the 
short-term, we plan to investigate how to focus an a type of information, graph 
or features, when it gets the upper hand. 
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